An Evaluation of Novelty and Diversity Based on Fuzzy Logic

Information retrieval systems are based on an estimation or prediction of the relevance of documents for certain topics associated to a query or, in the case of recommendation systems, for a certain user profile. Most systems use a graded relevance estimation (a.k.a. relevance status value), that is, a real value r(d, τ ) ∈ [0, 1] for the relevance of document d with respect to topic τ . In retrieval systems based on the Probability Ranking Principle [9], this value has a probabilistic interpretation, that is, r(d, τ ) is equivalent (in rank) to the probability that a user will consider the document relevant. We contend in this paper for an alternative interpretation, where the value r(d, τ ) is considered as the fuzzy truth value of the statement “d is relevant for τ”. We develop and evaluate two measures that determine the quality of a result set in terms of diversity and novelty based on this fuzzy interpretation.